[docs] Add docs for reworked Fivetran API #26026
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Nov 20, 2024
Merged
[5/n][dagster-fivetran] Implement
FivetranWorkspaceData to FivetranConnectorTableProps method
#25797
Merged
Open
This was referenced Nov 20, 2024
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
f42cebd
to
c01caf8
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
a0b8000
to
6aa71f2
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
c01caf8
to
9e707b8
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
6aa71f2
to
7a4fa77
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
9e707b8
to
d14c6d2
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
7a4fa77
to
3c894d1
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
d14c6d2
to
ad961a8
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
3c894d1
to
b2a5a00
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
ad961a8
to
66a92e1
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
b2a5a00
to
37b15c7
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
66a92e1
to
7cc3bc7
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
37b15c7
to
937ca4a
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
3d3a4ff
to
047b33d
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
de0b894
to
489b75a
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
047b33d
to
dea1be3
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
489b75a
to
bc55b9b
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
dea1be3
to
27ed743
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
bc55b9b
to
e7e3d18
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
27ed743
to
bf54e46
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
e7e3d18
to
a7794fb
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
bf54e46
to
68a881e
Compare
maximearmstrong
force-pushed
the
maxime/move-fivetran-docs-to-legacy-page
branch
from
a7794fb
to
3dd795c
Compare
maximearmstrong
force-pushed
the
maxime/add-new-fivetran-docs
branch
from
68a881e
to
e67667f
Compare
dpeng817
reviewed
Dec 3, 2024
| yield from fivetran.sync_and_poll(context=context) | ||
|
|
||
|
|
||
| # Alternatively, when creating all assets definitions for the Fivetran workspace |
There was a problem hiding this comment.
grammar here seems a bit off
dpeng817
reviewed
Dec 3, 2024
|
|
||
| fivetran_specs = load_fivetran_asset_specs(fivetran_workspace) | ||
| defs = dg.Definitions( | ||
| assets=[*fivetran_specs], resources={"fivetran": fivetran_workspace} |
There was a problem hiding this comment.
should be able to just fivetran_specs, no need for splat operator right
dpeng817
reviewed
Dec 3, 2024
| ) | ||
|
|
||
|
|
||
| # Creating assets definition for a given connector using the `@fivetran_assets` decorator |
There was a problem hiding this comment.
I think this comment is a bit redundant
dpeng817
requested changes
Dec 3, 2024
There was a problem hiding this comment.
Python files have hyphenated names, let's fix that.
Additionally, I think the structuring here is a bit weird. We should probably lead with build_fivetran_assets_... since that's the most straightforward way. Then, you can talk about "customizing execution functionality" and go through using the other APIs.
A few additional comments as well.
dpeng817
reviewed
Dec 3, 2024
|
|
||
| By default, Dagster will generate asset specs for each Fivetran asset and populate default metadata. You can further customize asset properties by passing a custom <PyObject module="dagster_fivetran" object="DagsterFivetranTranslator" /> subclass to the <PyObject module="dagster_fivetran" method="load_fivetran_asset_specs" /> function, the <PyObject module="dagster_fivetran" method="fivetran_assets" /> decorator or the <PyObject module="dagster_fivetran" method="build_fivetran_assets_definitions" /> factory. | ||
|
|
||
| ```python file=/integrations/fivetran/customize-fivetran-asset-defs.py |
There was a problem hiding this comment.
This whole code snippet feels really long, can we make use of startbefore and endafter to trim this down to only the relevant bits?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
As title
How I Tested These Changes
See preview